FORMAL GEOMETRIC QUANTIZATION 
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Abstract. Let X be a compact Lie group acting in an Hamiltonian way on 
a symplectic manifold (M, Q) which is prequantized by a Kostant-Souriau line 
bundle. We suppose here that the moment map <3? is proper so that the reduced 
space M/j, := ^~^{K ■ li)/K are compact for all fi. Following Weitsman |33| . 
we define the "formal geometric quantization" of M as 

The aim of this article is to study the functorial properties of the assigment 
M — . Q^°°(M). 
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1. Introduction and statement of the results 

Let M be an Hamiltonian if-manifold with symplectic form Q and moment map 
<I> : M — > t* . We assume the existence of a if-equivariant line bundle L on Af with 
connection of curvature equal to —ilu. In other words M is pre-quantizable in the 
sense of [20' and we call L the Kostant-Souriau line bundle. 

In the process of quantization one tries to associate a unitary representation of 
K to the data (Af, SI, <&, L). When M is compact one associates to these data a 
virtual representation Qk{M) € R{K) of K defined as the equivariant index of a 
Dolbeault-Dirac operator : Qk{M) is the geometric quantization of M . 

This quantization process satisfies the following functorial properties : 

[PI] When N and M are respectively pre-quantized compact Hamiltonian Ki 
and K2 manifolds, the product M x TV is a pre-quantized compact Hamiltonian 
Ki X iir2-manifold, and we have 

{l-l) QK,y.KAMxN) = QKAM)®QKAN) in R{KixK2)~R{Ki)®R{K2). 

[P2] li H <Z K is 'A connected Lie subgroup, then the restriction of Qk{M) to 
H is equal to Qh{M). 
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Note that [PI] and [P2] gives the following functorial property : 

[P3] When N and M are pre-quantized compact Hamiltonian if-manifold, the 
product M X N is SL pre-quantized compact Hamiltonian ii'-manifold, and we have 
Qk(M X A^) = Qk{M) ■ Qk{N), where • denotes the product in R{K). 

One other fundamental property is the behaviour of the iiT-multiplicities of 
Qk{M) that is known as "quantization commutes with reduction". 

Let T be a maximal torus of K, C'k C t* be a Weyl Chamber, A* C t* be the 
weight lattice, and K — A* D Ck be the set of dominant weights. The ring of 
character R{K) as a Z-basis G K : is the irreducible iiT-representation 

with highest weight /i. 

For any fi d K which is a regular value of $, the reduced space (or symplectic 
quotient) := <i>-^{K ■ fi)/K is an orbifold equipped with a symplectic struc- 
ture f]^. Moreover :— (L|$-i(^) 'C-^)/Kp, is a Kostant line orbibundle over 
(M^, ri^). The definition of the index of the Dolbeault-Dirac operator carries over 
to the orbifold case, hence Q(M^) G Z is defined. In [25], this is extended further to 
the case of singular symplectic quotients, using partial (or shift) de-singularization. 
So the integer Q(M^) G Z is well defined for every fi € K : in particular Q{M^) = 
if At ^ 3'(M). 

The following Theorem was conjectured by Guillemin-Sternberg |16| and is known 
as "quantization commutes with reduction" [2ll [25l [30l [2^ . For complete references 
on the subject the reader should consult [29] [32]. 

Theorem 1.1. (Meinrenken, Meinrenken-Sjamaar). We have the following equal- 
ity in R{K) 

Suppose now that M is non-compact but the moment map $ : A/ ^ i* is 
assumed to be proper (we will said simply " M is proper"). In this situation the 
geometric quantization of M as an index of an elliptic operator is not well defined. 
Nevertheless the integers Q{M^),^ G K are well defined since the symplectic quo- 
tient are compact. 

Following Weitsman [3^, we introduce the following 

Definition 1.2. The formal quantization of{M, H., $) is the element of R^°° {K) := 
homz(i?(iir), Z) defined by 

A representation E of K is admissible if it as finite if-multiplicities : 
dim(homx(V^, £^)) < oo for every ^ E K. Here R~°°(K) is the Grothendieck 
group associated to the K- admissible representations. We have a canonical inclu- 
sion i : R{K) ^ R-°°{K) : to F G R{K) we associate the map i{V) : R(K) Z 
defined hy W ^ dim(homi<-(y, W)). In order to simplify the notation, i{V) will be 
written V . Moreover the tensor product induces a structure of R{K)-m.oAn\e on 
R~°°{K) since E(S)V is an admissible representation when V and E are respectively 
finite dimensional and admissible representation of K. 
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It is an easy matter to see that [PI] holds for the formal quantization process 
Let N and M be respectively pre-quantized proper Hamiltonian Ki and 
K2 manifolds : the product M x A'^ is then a pre-quantized proper Hamiltonian 
Ki X ii'2-manifold. For the reduced spaces we have {M x -/V)(^j^^2) — ^t^i ^ -^a'2j 
for all Hi G Ki, ^2 G It follows then that 

(1.2) QkTxkSM ^N)^ QkT(m)^ QkT(^) 

in R-°°{Ki X K2) ~ R-°°{Ki)d)R-°°iK2). 

The purpose of this article is to show that the functorial property [P2] still holds 
for the formal quantization process Q~°°. 

Theorem 1.3. Let M be a pre-quantized Hamiltonian K-manifold which is proper. 
Let H C K be a connected Lie subgroup such that M is still proper as an Hamilton- 
ian H -manifold. Then Q~^°°{M) is H -admissible and we have the following equality 
in : 

(1.3) Q7i°^{M)\„ = Qh^{M). 

For H ^ K and z/ e we denote Nj^ = dim(hom^f (V^/^, V^^|h)) the multiplicity 
of Vj^ in the restriction V^^Ih- In the situation of Theorem ll.31 the moment maps 
relative to the K and H actions are and = P ° ^k, where p : — *■ f)* is 
the canonical projection. 

Corollary 1.4. For every v g H , we have : 

(1-4) Q (M,,ff) = KQ{M^.k) ■ 

where Mj^^h = ^^{H ■ v)/H and M^^k = ^^{K ■ fJi)/K are respectively the 
symplectic reductions relative to the H and K-actions. 

Since is equal to the X-quantization of the "quantization commutes 

with reduction" Theorem tells us that Nj^ — Q{{K ■ h)^,h) ■ in particular A^/f ^ 
implies that v G p{K ■ fi) ■^=^ fi € K- p~^(i^). Finally 

KQiM^,K)^0 =^ l^eK-p-\X) and <P],''{p) ^ >i. 

Theses two conditions imply that we can restrict the sum of RHS of p.4p to 

(1.5) /xei^n$K(i^-$^i(i^)) 

which is finite since is proper. 

Theorem 11.31 and (II. 2p gives the following extended version of [P3]. 

Theorem 1.5. Let N and M be tiuo pre-quantized Hamiltonian K-manifold where 
N is compact and M is proper. The product M x N is then proper and we have 
have the following equality in R~°°{K) : 

(1.6) Q-K^{M xN) = Q-^°-{M)-Qk{N) 

For H,X,e e K we denote C^g = dim(homjj(V'^^, V^^ Vg^)) the multiplicity 
of V^f in the tensor V-^ ® . Since ® Vg^ is equal to the quantization of 
the product K-X x K-9, the "quantization commutes with reduction" Theorem 
tells us that C'^g — Q{{K ■ X x : in particular C^g 7^ implies that 

W l|All<||01l + il/il|. 
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Corollary 1.6. In the situation of Theorem \1.5\ we have for every fj, ^ K : 
(1-7) Q m X N)^) = ^ C,%Q (Af,) Q (Ne) . 

Since N is compact, Q{Ng) 7^ for a (**) e {finite set}. Then (*) and (**) 
sliow tliat the sum in the RHS of (|1.7p is finite. 

Weistman [33] studied the formal quantization procedure in the case of K = 
U(n). Using a method of symplectic cutting [211 [35] he defines for two increasing 
sequence of positive integers rn,Sn a family of cut-spaces Af"'**" which are com- 
pact. Under the hypothesis that the cut spaces iVf""'*" are smooth, he notes that 
Q].°°{M) = lim Qk{M''"''"), and he was then able to show Theorem [TSj 

n — *oo 

Our proof of Theorem 11.31 uses also a technique of symplectic cutting but which 
is valid for any compact Lie group actions. We have to overpass the difficulties 
concerning the non-smoothness of the cut-spaces. For this purpose we introduce 
another method of symplectic cutting that uses the wonderfull compactifications of 
Concini-Procesi |131 114j , and we prove an extension of the "quantization commutes 
with reduction" Theorem to the singular setting. 

Acknowledgements : I am grateful to Michel Brion and Nicolas Ressayre 
for enlightening discussions about the wonderfull compactifications. I thanks also 
Anton Alekseev for these remarks on a preliminary version of the paper. 

2. Quantization commutes with reduction 

In this section we precise the definition of the quantization of a smooth and 
compact Hamiltonian manifold. We extend the definition to the case of a singular 
Hamiltonian manifold and we prove a "quantization commutes with reduction" 
Theorem in the singular setting. 

In the Kostant-Souriau framework, an Hamiltonian iiT-manifold (Af, f2j\/, ^m) is 
pre-quantized if there is an equivariant Hermitian line bundle L with an invariant 
Hermitian connection V such that 

(2.8) /:(A)-Vx„ =i($M,A} and ^ -z^m, 

for every A e J. Here Am is the vector field on A/ defined by Am(to) = ^^"''^"^lo- 
(L, V) is also call a Kostant-Souriau line bundle. Remark that the conditions 
(12. 8p implies through the equivariant Bianchi formula the relation 

(2.9) t(AM)r!M = -d($M,A), Xet 
We will now recall the notions of geometric quantization. 

2.1. Geometric quantization : the compact and smooth case. We suppose 
here that {M, ^Im, <I>m) is compact and is prequantized by a Hermitian line bundle 
L. Choose a A-invariant almost complex structure J on M which is compatible 
with Cli^i in the sense that the symmetric bilinear form J-) is a Riemannian 

metric. Let Ol be the Dolbcault operator with coefficients in L, and let dj^ be 
its (formal) adjoint. The Dolbeault-Dirac operator on M with coefficients in L is 
Dl +dl, considered as an operator from A°'''^'='^{M, L) to y4°'°<*<^ (A/, L). 
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Definition 2.1. The geometric quantization of{M, VIm, ^m) is the element Q/f (A/) S 
R{K) which is defined as the equivariant index of the Dolbeault-Dirac operator Dl- 

Remark 2.2. • We can define the Dolbeault-Dirac operator D[ for any invariant 
almost complex structure J. If Jq and Ji are equivariantly homotopic the indices 
of D^' and D 'j^ coincides (see [28] ). 

• Since the set of compatible invariant almost complex structure on M is path- 
connected, the element Qk{M) S R{K) does not depend of the choice of J . 

2.2. Geometric quantization : the compact and singular case. We are in- 
terested to defined the geometric quantization of singular compact Hamiltonian 
manifolds : here "singular" means that the manifold is obtain by symplectic reduc- 
tion. 

Let (Nj^In) be a smooth symplectic manifold equipped with an Hamiltonian 
action oi K x H : we denote ($k, $h) : iV — > J* © ()* the corresponding moment 
map. We assume that N is pre-quantized hy a, K x i/-equivariant line bundle L 
and we suppose that the map is proper. One wants to define the quantization 
of the (compact) symplectic quotient 

N//„H:=<P-h\0)/H. 

When is a regular value of ^h, N // qH is a compact symplectic orhifold 
equipped with an Hamiltonian action of K : the corresponding moment map is 
induced by the restriction of <^k to (f>^^(0). The symplectic quotient N // is 
pre-quantized by the line orbibundle 

Definition [2A] extends to the orbifold case, so one can still defined the quantization 
of N//^H as an element Qk{N//qH) e R{K). 

Suppose now that is not a regular value of ^h- Let Th be a maximal torus of 
H , and let Ch C be a weyl chamber. Since is proper, the convexity theorem 
says that the image of intersects Ch in a closed locally polyhedral convex set, 
that we denoted Ah{M) [22] . 

We consider an element a E AniM) which is generic and sufficiently closed 
to S Ah{M) : we denote Ha the subgroup of H which stabilizes a. When 
a € Ah{M) is generic, one can shows (see l25l) that 

Nl/^H:=<^-^Ha)/Ha 

is a compact if-Hamiltonian orbifold, and that 

La /Ha. 

is a if-equivariant line orbibundle over N // ^H : we can then define like in Definition 
12. li the element Qk {N // ^^H) e R{K) as the equivariant index of the Dolbeault-Dirac 
operator on N // ^H. 

Proposition-Definition 2.3. The elements Qk{N // ^H) G R{K) do not depend 
of the choice of the generic element a G Ah{M), when a is sufficiently closed to 0. 
The common value will be taken as the geometric quantization of N//gH, and still 
denoted Qk{N//^H). 

Proof. When TV is compact and K = {e}, the proof can be founded in [25] and 
in [28]. The K-theoretic proof of [28] extends naturally to our case. □ 
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2.3. Quantization commutes with reduction : the singular case. In section 
\2.2\ we have defined the geometric quantization Qk{N // ^H) G R{K) of a com- 
pact symplectic reduced space N // ^^H . We will compute its multiplicities like in 
Theorem 11.11 

For every /i € If, we consider the coadjoint orbit ■ /i ~ KjK^^ which is 
pre-quantized by the line bundle C[^] ~ K Xk^ 'C^. We consider the produclQ 
N xK ■ fjL which is an Hamiltonian K x H manifold which is pre-quantized hy K x H- 
equivariant line bundle L (g) C|~|. The moment map N x K ■ ii i* x i)* , {n, ^) 
(<I>k("-) — is proper, so the reduced space 

{N//,H)^ {N X irjl)//^, ,^K x H 

is compact. Following Proposition 12.31 we can then define its quantization 
Q{{N // f^H) ^) E Z. The main result of this section is the 

Theorem 2.4. We have the following equality in R{K) : 

(2.10) Qk{N/UH) = J2 QiiN/UH)^)V^. 

Proof. The proof will occupied the remaining of this section. The starting point 
is to state another definition of the geometric quantization of a symplectic reduced 
space which uses the Atiyah-Singcr's theory of transversally elliptic operators. 

2.3.1. Transversally elliptic symbols. Here we give the basic definitions from the 
theory of transversally elliptic symbols (or operators) defined by Atiyah-Singer in 
[1] . For an axiomatic treatment of the index morphism see Berline-Vergne [H [9] 
and for a short introduction see pS] . 

Let A:" be a compact Ki x _R'2-ma'nifold. Let p : TA" — > A" be the projec- 
tion, and let {—,—)x be a Ki x iir2-invariant Riemannian metric. If E^,E^ are 
Ki X iir2-equivariant vector bundles over X, a Ki x ii'2-equivariant morphism 
cr G r(TA',hom(p*£;°,p*£;i)) is called a symbol. The subset of aU {x,v) G TX 
where a{x,v) : — > E], is not invertible is called the characteristic set of cr, and 
is denoted by Char(cr). 

Let Tkj X be the following subset of l^X : 

Tk,X = {{x, v) G TAf, {v, Xm{x))^, = for aU X G ^2} • 

A symbol cr is elliptic if cr is invertible outside a compact subset of TX (i.e. 
Char((T) is compact), and is K2-transversally elliptic if the restriction of a to T/^jA" 
is invertible outside a compact subset of ^k^"^ (i-6- Char((T) n T/^^ A" is compact). 
An elliptic symbol a defines an element in the equivariant K-theory of TA" with 
compact support, which is denoted by \^Ki-kK2 (TAf), and the index of cr is a virtual 
finite dimensional representation of Ki x K2 [SI SI [3 [Sj ■ 

A K2-transversally elliptic symbol cr defines an element of K^i x/4'2 (T/fa-^^); ^'I'i 
the index of cr is defined as a trace class virtual representation of Ki x K2 (see 
[T] for the analytic index and [51 [S] for the cohomological one) : in fact Index'* (cr) 
belongs to the tensor product R{Ki)®R~°° {K2) . 



K ■ n denotes the coadjoint orbit with the opposite symplectic form. 
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Remark that any elliptic symbol of TX is /'ir2-transversally elliptic, hence we have 
a restriction map ^ (T'-f ) (Ti^a"^): ^^'^ ^ commutative diagram 

(2.11) Ka-ixA-.ITA-) ^Kk,xkAT^k,X) 



Index'^ 



Index'* 



Using the excision property, one can easily show that the index map Index^ : 
(T^aW) R{Ki)(S)R~°^ {K2) is still defined when U is a KiX iir2-invariant 
relatively compact open subset of a Ki x i4r2-manifold (see [28][section 3.1]). 

2.3.2. Quantization of singular space : second definition. Let {X, ilx) be an Hamil- 
tonian Ki x _R'2-nianifold pre-quantized by a Ki x _R'2-equivariant line bundle L. 
The moment map $2 : ^ ^2 relative to the i4r2-action is supposed to be proper. 
Take a compatible Ki x iir2-invariant almost complex structure on X. We choose a 
Ki X ii'2-invariant Hermitian metric on the tangent bundle TX, and we iden- 
tify the cotangent bundle T*X with TX. For {x,v) G TX, the principal symbol 
of the Dolbeault-Dirac operator Ol +9^ is the Clifford multiplication cx{v) on the 
complex vector bundle A*TxX(^Lx- It is invcrtiblc for w 7^ 0, since cx{v)'^ = — 

When X is compact, the symbol cx in elliptic and then defines an element of the 
equivariant K-group of TX. The topological index of cx G Kxixa:2(TA:') is equal 
to the analytical index of the Dolbeault-Dirac operator + d\ : 

(2.12) Qkixk.(^) -lndex-^(c^) in R{Ki)®R{K2). 

When X is not compact the topological index of c;^ is not defined. In order 
to give a topological definition of QkA'^^ // ()K2), we will deform the symbol cx as 
follows. Consider the identification 63 — ^2 defined by a iir2-iiivariant scalar product 
on the Lie algebra I2 ■ From now on the moment map $2 will take values in 62 : a-nd 
we define the vectors field on X 

(2.13) = ($2(a;))Af(a;), x(^X. 
We consider now the symbol 

c^(u) = c(w - K:r), v^T,,X. 

Note that c^(u) is invertible except ii v — Kx. If furthermore v belongs to the 
subset TifjM of tangent vectors orthogonal to the _fir2-orbits, then w = and 
Kx = Q. Indeed tangent to K2 ■ x while v is orthogonal. 

Since k, is the Hamiltonian vector field of the function ||3'2|P, the set of zeros of 
K coincides with the set Cr(||$2|P) of critical points of ||$2p. 

Let U G X he a Ki x i^2-invariant open subset which is relatively compact. If the 
border dU does not intersect Cr(||<I>2|P), then the restriction c')^\u defines a class 
in Ka'ixK2(Ta'2^) since 

Char(c^|;^) n Tk.U ~ Cr(||$2f ) H U 

is compact. In this situation the index of Cfi\u is defined as an element Index'^(cK|w) G 
R{Ki)®R-°^{K2). 

Theorem 2.5. The K2-invariant part 0/ Index^(c^ |iy) is equal to : 
• QkA^ //0K2) when $2-1(0) C U, 
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• in the other case. 

Proof. When Ki — {e}, the proof is done in [55] (see section 7). This proof 
works equaUy well in the general case. 

Remark 2.6. // X is compact we can take lA — X in the last Theorem. In this 
case the symbols c'^ and Cx defines the same class in K.KixK2{'^'^) so they have 
the same index. Theorem \2.5\ corresponds then to the traditional "quantization 
commutes with reduction" phenomenon : [QKixK-ii-^)]^^ ~ QkA-^ // QK2). 

^From now one we will work with this topological definiton for the geometric 
quantization of the reduced i^i-Hamiltonian manifold X jj (which is possibly 
singular): Q,Kx{-^ II q^^-i) = [Index'^(c^|jY)]'^^ where U is any relatively compact 
neighborhood of $^^(0) such that dU fl Cr(||<i>2|p) = 0. The functorial properties 
still holds in this singular setting. In particular : 

[P2] If C iCi is a connected lie subgroup, then the restriction of Qk^ {X H ^K-i) 
to B is equal to Q,h{X I ^^Ki) . 

2.3.3. Proof of theorem \2.4\ We come back in the situation of sections [2.21 and [2?3l . 

First we apply Theorem [23] to X ^ N , Ki = K anA K2 = H . ([210]) is trivially 
true when ^ Image($//). So we suppose now that G Image($if), and we 
consider an x iJ-invariant open subset lA <Z N which is relatively compact and 
such that 

$ff^(0)cW and aZ^nCr(||$Hf ) = 0. 

We have Qk{N H ^H) = [Index'^(c^" \u)]^ and one want to compute its K- multi- 
plicities TO^,/i € K. Here is the vectors field on N associated to the moment 
map <^H (see ([2T3| ). 

Take n £ K. We denote the principal symbol of the Dolbeault-Dirac opera- 
tor on K ■ fj, with values in the line bundle C[_^] : we have Index'^'''(c_ju) = (V^^)*. 

We know then that the multiplicity of [Index^(c^" \u)]^ relatively to V^^ is 
equal to 



with V = U X K ■ ^. This identity is due to the fact that we have a "multiplication' 



so that Index'^''-^-^(CTi 0(72) = Index" (cti )• Indcx-^'^ (0-2) in R-°°{K x H). See p. 

Consider now the case where X = N x K ■ ^j,, Ki = {e} and K2 = K x H. After 
Theorem I2.5[ we know that 




tKxH 



(2.14) 



KA-xH(TffW) X Kk{T{K-h)) 

(o-l,cr2) 



CTl (72 • 



QiiNll.H)^) ^ Index^(c:^|v) 



n KxH 



(2.15) 



where k is the vector field on N x K ■ 11 associated to the moment map 



$ : iV X — > r X f)* 

(x,o ^ (<i>K(-^)-e,*H(")) 

Note that V = W x A' • /i is a neighborhood of $"^(0) C ($h)"^(0). 
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Our aim now is to prove that the quantities (|2.14p and (|2.15|) are equal. 

Since the definition of k needs the choice of an invariant scalar product on the 
Lie algebra 4 x f), we will precise its definition. Let || • \\k and || • \\h be two invariant 
Euclidean norm respectively on 6 and f). For any r > we consider on ? x f) the 
invariant Euclidean norm r)||^ = + 

Let be the vector field on N x K ■ fi associated to the map N x K ■ 
6*, (x,^) 1-^ ^k{x) — and where the identification t ~ 6* is made through the 
Euclidean norm \\-\\k (see (|2.13p '). For (x,^) £ NxK ■ /i, we have the decomposition 



Let ft" be the vector field on N x K ■ fj, associated to the map N x K ■ ii 
6*, (x,^) ^-^ ^h{x), and where the identification f) ~ f)* is made through the Eu- 
clidean norm || • \\h- For (x, ^) E N x K ■ fj,, we have the decomposition 

k"{x,0 = {k"{x),0) e T^NxT!=K-n. 



For any r > 0, we denote by Hr the vector field on N x K ■ fi associated to the 
map <&, and where the identification 6 x (j ~ 6* x [}* is made through the Euclidean 
norm || • We have then 

Kr = k" -\- r k'^ 

Now we can precise (|2.15p . Take an invariant relatively compact neighborhood 
U of $^^(0) such that dU fl {zeros of k"} = 0. With the help of a invariant 
Riemannian metric on X we define 

Eh = inf ||K"(a;)|| >0 and ek ^ sup ^)||. 

Note that for any < r < we have dl4 x K-jin {zeros of k" + tki} = ^, and 
then dV n {zeros of k^} = for the neighborhood V :— U x K ■ oi <i>~^(0). We 



can then use Theorem 12.51 : for < r < — we have 

' ' sk 

Q((7V//oi?)^) = [index^(c^'-lv)]''''' 

We are now close to the end of the proof. Let us compare the symbols c'^\v and 
c^" \u in Kxxff(Ti<-xff x K-fj.)). First one sees that the symbols cx is 
equal to the product c^v hence the symbols c^" \u is equal to c'^\v 
when r = 0. Since for t < ^ the path s £ [0, r] c^^lv defines an homotopy of 
K X i7-transversally elliptic symbols on V, we get 

Index^(c;^'-|v) = Index^(c^'' \u c_^) 

and then = Q{{N // ^^H) ^) . □ 



3. Wonderful compactifications and symplectic cutting 



In this section we use the wonderful compactifications of Concini-Procesi [T3t [T4] 
to perform symplectic cutting. 
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3.1. Wonderful compactifications : definitions. We study here the wonderful! 
compactifications from the Hamiltonian point of view. 

We consider a compact connected Lie group K and its complexification K^. Let 
T be a maximal torus of K , and let W := N{T)/T be the Weyl group. Consider t*, 
the dual of the Lie algebra of T, with the lattice A* of real weights. Let Ck C t* 
be a Weyl chamber and let if A* n Ck be the set dominants weights. 

Definition 3.1. A polytope P in t* is K-adapted if : 

i) the vertices of P are regular elements of A* , 

ii) P is W -invariant, 

iii) P is Delzant. 

Example : When K a,s a trivial center, the convex hull of ■ /i is a if-adapted 
polytope for any regular dominant weight /z. 

Proposition 3.2. There exists K-adapted polytopes in t*. 

Proof. Let us use the dictionary between polytopes and projective fan [77] . 
Conditions ii) and iii) of Definition 13.11 means that we are looking after a smooth 
projective M^-invariant fan T in t. Condition i) means that each cone of T of 
maximal dimension should not be fixed by any element of \ {Id}. For a proof of 
the existence of search fan, see [II1II2]- In particular condition (*) in Proposition 
2 of [12] implies i). □ 

In the rest of this section, we consider a if -adapted polytope P. Let [P]i be 
the union of all the closed facet of dimension 1 : we label the elements of [P] i fl if 
by {Ai,--- ,AAr}. Since P is if -adapted, when is a vertex of P there exits 

, • ■ • , ckj,. belonging to HA* = VF- {Ai, • • ■ ,Xn} such that aj-^ — Ai, • • ■ , aj^ — 
Ai is a basis of the lattice A*. 

Let V\. be an irreducible representation of if with eighest weight Ai: these 
representations extend canonically to the complexification ifc. We denote p : ifc — > 
^iLiGL{V\-) the representation of ifc on V := (BiLiVxi- We consider the vector 
space 

E ^ ®fLi^nd{VxJ 

equipped with the action of ifc x ifc given by : (51,32) • / = Pidi) ° f ° p{92)~^- 
Let ¥{E) the projective space associated to E : it is equipped with an algebraic 
action of the reductive group ifc x ifc. We consider the map g i~> [p{g)] from ifc 
into ¥{E), that we denote p. 

Lemma 3.3. The map p : ifc ^ ¥{E) is an embedding. 

Proof. Let g & ifc such that p{g) = [Id] : there exist a € C* such that 
p{g) = aid. The Cartan decomposition gives 

(3.16) p(k) = -^Id and p(e'^) = \a\Id 

\a\ 

for g — ke^-^ with k E K and Jf G t. Since there exist Y,Y' E t and u,u' K such 
that k = ue^u^^ and X — u' ■ Y' , (|3.16p gives 

(3.17) p{e^)^-^Id and p{e'^' ) = \a\Id. 
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Since each element of [P]i D A* = W ■ {Ai, • • • , Xn} is a weiglit for the action of 
Tc on ©iliVx,, ([3l7l) impUcs that for every a G [P]i n A* we have 

(3.18) e'<"'^> = ^ and e-<"-^'> = |a|. 

\a\ 

Since there exists ajg , ■ ■ ■ , aj^ G [P] i H A* such that aj-^ — Uj^ , • ■ ■ , otj^ — oij^ is a 



e 



basis of the lattice A*, (|3.18p imphes that F' = and that Y e ker(Z e t 
We have proved that g — e. □ 

Let Tc C Kc the complexification of the (compact) torus T C K. 

Definition 3.4. Let Xp he the Zarisky closure of p{Kc) in P{E) and let yp C Xp 

be the Zarisky closure of p{Tc) inP{E). 

Since p{Kc) — Kc x Kc ■ [Id] and p{Tc) = Tc x Tc • [Id] are orbits of algebraic 
group actions their Zariski closures coincide with their closures for the Euclidean 
topology. 

Theorem 3.5. The varities Xp and yp are smooth. 

The proof will be done in the next section 

3.2. Smootiiness of Xp and yp. Let i? be a complex vector space equipped with 
a linear action of a reductive group G. Let Z C P(i?) be a projective variety which 
is G-stable. We have the classical fact 

Lemma 3.6. • Z possess closed G-orbits. 

• Z is smooth if Z is smooth near its closed G-orbits. 



Proof. Let zq & Z and consider the Zariski closure G ■ zq C Z. If G • zo is not 
closed, we take zi £ G • zq \ G • ■ we have diniG • zi < dimG ■ zq. By induction 
we find a sequence zi, - ■ ■ , Zp with Zk+i £ G ■ Zk\G ■ Zk for k < p and G • Zp closed. 
For the second point, we have just to note that if Z is singular, the subvariety 
2;smg 2. oi singular points is G-stable and then contains a closed G-orbits. □ 

We are interested here respectively in 

• the Kc X Xc-variety Xp C P(£;) C P(End(V')) 

• the Tc X Tc-variety yp C P(£;). 

Since the diagonal Zc — {[t,t)\t G Tc} stabihzes [Id], its action on yp is trivial. 
Hence we will restrict ourself to the action of Tc x Tc/Zc — Tc on yp : for t € Tc 
and [y] £ yp we take t ■ [y] — [p{t) o y]. 



3.2.1. The case of yp. We apply Lemma [3?6l to the Tc-variety J^p = Tc ■ [Id] 
in P{E). Let {aj,j G J} be the Tc weights on (©^iVa.,p) counted with their 
multiphcity. Their exists a orthonormal basis {vj,j G J} of (BfLi^Xi such that 
Id = J2i<£j ""j ® ^7 ^^'^ 



(3.19) p(e^) =^e*<"-^>?;, 01-*, Z etc. 

So the action of G Tc on [Id] G P{E) is . [Id] = J^jeJ e'^^^'^^Vj (g) v* . We 
introduce a subset J' of J such that for every j e J there exists a unique j' G J' 
such that aj = aj'. So the variety yp belongs to ¥{E') where E' = (Bj'ej'Cnij' 
with nij' = J2j aj=a -, ""j ® ^1 • The closed Tc orbits in P{E') are [m^'], j' g J. 
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Lemma 3.7. [mj^] G yp if and only if aj^ is a vertex of the polytope P. 

Proof. If aj^ is a vertex of P, there exists X G t such that (aj^, X) > (aj, X) 
whenever aj^ ^ aj. Hence e'^"'^ ■ [Id] tends to [mj^] when s — > +00. If aj^ is not 
a vertex of P, there exist L <Z J' \ {jo\ such that aj^ — X]/eL o,ioii with < a/ < 1 
and ai — 1. So yp belongs to the closed subset defined by 

[ ^ 5j>mf] e V{E') such that n/gi|(5/|°' = \6jJ. 
j'&J' 

Hence [rn^-J <^yp. a 

Remark 3.8. When aj is a vertex of the polytope P , the multiplicity of aj in 
(Sf^^iVx^ is equal to one, so ruj = Vj (g) v* . 

Consider now a vertex aj^ of P (for jo G J'). We consider the open subset 
V C P(i?') defined by Ej'ej' (^j'TO^/] G V <^ 7^ 0, and the diffeomorphism 

i/j : V ^ <C'^'\{^°^ ^ [^jifzj' ^j'TUji] I— > {j-)j'^ja- The map 1/; realizes an algebraic 
diffeomorphism between yp n V and the affine subvariety 

The set of weights aj , j G J contains all the lattice points that belongs to the one 
dimensional faces of P. Since the polytope P is if-adapted, there exists a subset 
Lj^ C J' such that ai — Uj^, I G Lj^ is a Z-basis of the group of weights A*. And 
for every j' ^ jo we have 

(3.20) a-j' - a-j^ = ^ n^j>iai - a-jj with G N. 

We define on C^^° the monomials Fj'(Z) = Hzgi^.^ (Z;)"^' . Note that P,'(Z) = 
when j' ~ I E Lj^ . Now it is not difficult to see that the map 

C^^o , £J'\{3o} 

realizes an algebraic diffeomorphism between C^^° and Z. 

Finally we have shown that yp is smooth near [mjj : hence yp is a smooth 
subvariety of F{E). Since Tc acts on yp with a dense orbit, J^p is a smooth 
projective toric variety. 

3.2.2. The case ofXp. Let E := ®,^iEnd(V"A, ). The closed A'c x i^c-orbit in ¥{E) 
are those passing through [v\. ® v\] where v\. G Vx^ is a highest weight vector 
(that we take of norm 1 for a i^- invariant hermitian structure). 

Lemma 3.9. [v\^ ® v\] G Xp if and only if Xi is a vertex of the polytope P. 

Proof. If is a vertex of P, we have proved in Lemma [3.71 that [vx^ w^.] 
belongs to yp and so belongs to Xp. We prove the converse in Corollarv l3.f 41 □ 

For the remaining of this section we consider a vertex A^^ G -ftT of the polytope 
P. Let _B+, B" the Borel subgroups fixing respectively the elements [vx^ ] G P{Vxi) 
and [v1. ] G P{y^.). Consider also the unipotent subgroups C fixing 
respectively the elements vxi & Vx^ and w^. G Vy. 



FORMAL GEOMETRIC QUANTIZATION 



13 



We consider the open subset Vsnd C P{E) of elements [/] such that 
(^A, J fi'v>^^a )) ^ ■ "'^End is X B+ stable. Consider the open subset V C TP{V\^^ ) 
andV* C pVa* )) defined by : 

• [v]eV ^ {vl^ ,v) ^0 : V is B- stable, 

• [C] e V* ^ {CvxJ ^ : V* is 5+ stable. 

We consider now the rational maps / : P{E) --^ )if ^ fi'^Xi ) and r : 

P(£') P(V;^* ), / 1-^ wj. of. The map / and r are defined on VEnd : they defined 
respectively i3^-equi variant map from VEnd into V, and _B+-equivariant map from 
VEnd into V*. 

The orbits Kc ■ wa,„ C P(Vxi„ ) and Kc ■ v^. C P(F^* ) are closed and we have 
Kc ■ vx,^ n V = ■ WA,„ ^ N~ 

The rational map {l,r) : P{E) P{V\^^) x P(y^* ) induced then a map q : 
VEnd n Xp X which is x A^+-equivariant : 

for X e VEnd n A'p, and G A^='=. 

We can now finish the arguments. The set N^T^N^ C Kc is dense in Kc, so it 
is now easy to see that the map 

A^- X X n VEnd Xpn VEnd 

is a diffeomorphism. We have proved previously that yp n VEnd is a smooth affine 
variety, hence Xp is smooth near the closed orbit Kc x Kc ■ [wa; "X) w^.] C X r\ VEnd- 
Lemma 13.61 tells us then that Xp is smooth. 



3.3. Hamiltonian actions. First consider an Hermitian vector space V . The Her- 
mitian structure on End(F) is {A, B) := Tr(A_B*), hence the associated symplectic 
struture on End(y) is defined by the relation i}^ndiA,B) := -Im(Tr(yli?*)). 

Let U{V) be the unitary group. Let u{V) be the Lie algebra of U{V). We will 
use the identification e : u{V) ^ u{V)* , X ^ ex where ex{Y) = -Tr(Xy). The 
action U{V) x U{V) on End(F) is [g^h) ■ A = gAh^^. The moment map relative 
to this action is 

End(F) — > u{V)* X u{V)* 

A 1 — > ^{iAA*,^iA*A). 

We consider now the projective space P(End(T^)) equipped with the Fubini-Study 
symplectic form ftps- Here the action of U{V) x U (V) on P(End(V")) is hamiltonian 
with moment map 

P(End(P^)) — > n{V)* X n{V)* 
(iAA* -iA*A 

[A] 



where = Tr(ylA*) (see [lS][Section 7]). If p : A' U{V) is a connected Lie 
subgroup, we can consider the action of x if on P(End(V")). Let ttk ■ u(y)* ~* t* 
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be the projection which is dual to the inclusion p : t ^ The moment map 

for the action oi K x K on (P(End(y)), fJps) is then 

(3.21) p(End(y)) — > r X r 

^ -^MiAA*l-nK{iA*A)). 

We are interested here respectively in 

• the projective variety Xp C P(End(F)) with the action oi K y. K, 

• the projective variety yp C P(End(y)) with the action of T x T, 

where V = ©f^-^Vx^. The Fubini-Study two-form restrict into symplectic forms on 
Xp and yp. The action o{ K x K on Xp is Hamiltonian with moment map 

(3.22) ^KxK-.Xp rxr 

[x] I — > ---T2(7rii:(za;a;*), -TT/f (ia;*a;)). 
iFll 

Since the diagonal Z = {{t,t)\t E T} acts trivially on yp we restrict oursclf to 
the action of T x ~ T on yp. Let us compute the moment map $t : yp i* 
associated to this action. First we have 

\\y\r \\y\r 

where ttt ■ u{V)* i* is the projection which is dual to p : t — > n(V). Since 
p{X) = i J2jeJ ^ ^ small computation shows that for B e ii{V) ~ u{V)* 

we have tttIB) = —i^j^j{Bvj,Vj)aj. Finally for any \y] G yp we get 

Together with the action on T, we have also an action of the Weyl group W = 

N{T)/T on yp : iov w €W we take 

(3.24) w-[y] = [p{w)oyo p{w)-\ M G 

This action is well defined since the diagonal Z CT xT acts trivially on yp. The 
set of weights {aj,j e J} is stable under the action of W, hence it is an easy fact 
to verify that the map $r is W-equi variant. 

A dense part of yp is formed by the elements • [Id\ = [p{e^)]. Take Z = 
X + iy G tc. We have $T(e^ • [Id]) = i^riY) G t* with 

(3-25) MY) = ^ ,i2(.,..) E^"'^"-"^".- 

Hence the image of the moment map $t : yp — > t* is equal to the closure of the 
image of the map '^t : t — > t*. 

Proposition 3.10. The map tjjT realises a diffeomorphism between t and the inte- 
rior of the polytope P C t* . 



Proof. Consider the function Ft : t ^ R, Ft{Y) = In (^^^j e^""'^^ j . and let 
Lt : t — > t* be its Legendre transform : Lt{X) = dFT\x. Note that we have 
Lt{-2Y) = V'T(y). 
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We see that Ft is strictly convex. So, it is a classical fact that Lt realizes a 
difFeomorphism of t onto its image, and for ^ G t* we have 

C e Iniage(LT) ^ hm Ft{Y) - Y) = oo 

Y^oo 

^ lim y"e<"^-«'^> = oo. 
In order to conclude we need the following 

Lemma 3.11. Let £ J} be a sequence of elements of t* , and let Q he its 

convex hull. We have 

lim ^e^'^^-'^^ = oo G Interior((5) 

Proof. First we see that ^ Interior((3) if and only there exists f £ t — {0} 
such that {(3j,v} < for all j : for such vector v, the map t — » X^jeJ 
is bounded. Suppose now that \imY^ooJ2jej^^^''^^ Then there exists a 

sequence {Xk)k G t such that lim^ \Xk\ = oo and for all j the sequence ((/3j, Xk))k 
remains bounded. If t; is a limit of a subsequence of (y^^)j, we have then {(3j, v) < 
for all j. □ 

Lemma 3.12. For [y] e 3^p we have ^kxkHv]) = (*T([y]), ~<^T{[y]))- 

Proof. It's sufficient to consider the case y = p{e^) = ^j^j e^^'^^'^'^vj ® v*, for 
Z ^ X + iY etc. Then yy* = y*y ^ J2j e"2<"^'^>Wj «) w* = ^(e^''^). So it remains 
to prove that TrK{iyy*) = TTT{iyy*). We have to check that {TTK{iyy*), [U,V]) = 
for t/ G t and V £ i. We have 

{TTKiiyyldU^V]) = -iTv{yy*pi[U,V])) 

= -zTr(p(e2»^)[p(C/),p(F)]) 

□ 

Theorem 3.13. We have 

• Image($T) = P, 

• Image($if xk) = {(^i • ^, -fcz • ^ | ^ G P and fci, fcs G JsT}, 
. yp C <i>]^W(r X t*), 

• *J'KxK(™t6rior(C)) C 3^p,, where C ^ Ck 'x -Ck- 

Proof. The first point follows from Proposition l3.10l Since the map (fci ,t,k2) i-^ 
kitk2 from K x Tc x K into Kc is onto, we have 

(3.26) Xp = {KxK)-yp. 

So if [x] G Ap, there exist \y\ G 3^ and ki,k2 E K such that [x] = {ki,k2) ■ [y], hence 

'^kxkHx]) = iki,k2) ■ ^KxK{[y]) 

(3.27) = (A;i.$j,([2;]),-fc2.$j,([y])) 

The second point is proved. The third point follows also from the identity (|3.27p 
when ki = k2 = e. Consider now [x] — (fci, /C2) • [y] such that '^kxk{[x\) belongs to 
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the interior of the cone Ck x —Ck- Then ki ■ $T([y]) and k2 ■ ^T{[y\) are regular 
points of Ck- This iniphes that /ci, A;2 G N{T) and fc2^r^ '= ^- 

[x] = {ki,k2)-[y\ 

= (e,fc2fcr')- ((A:i,fci)-[y]) e^P 

since yp is stable under the actions of T x T and ly. □ 

Corollary 3.14. // [vx. ® w^.] G Ap i/ien is a vertex of the polytope P. 

Proof. Let x — vx-(E)v'^ , and suppose that [x] belongs to Xp. In order to show 
that [x] E yp, we compute ^kxkHx]). We see that xx* — x*x — x and ||a;|| = 1 
so ^kxk{[x]) — {iTKiix), ~TTK{ix)). For X E t we have 

{7rKiix),X) = -iTT{vx,®vlpiX)^ 
= i^X). 

We have then ^kxk{[x]) = {Xi, —Xi) with Xi beiing a regular point of Ck '■ hence 
[x] S yp. Now we can conclude with the help of Lemma [3.71 Since [vxi ® V^.] 
belongs to yp, the weight Xi is a vertex of the polytope P. □ 

Remark 3.15. In this section, Theorem \3.13\ was obtain without using the fact that 
the varieties Xp and yp are smooth. Hence Corollary \3.14\ can be used to prove 
the smoothness of Xp . 

3.4. Symplectic cutting. Let {M,Q,Mt^m) be an Hamiltonian A'-manifold. At 
this stage the moment map $m is not assumed to be proper. We consider also the 
Hamiltonian K x X-manifold Xp associated to a iiT-adapted polytope P. 

The purpose of this section is to define a symplectic cutting of M which uses 
Xp. The notion of symplectic cutting was introduced by Lerman in [21] in the case 
of a torus action. Later Woodward [321 extends this procedure to the case of a 
non-abelian group action (see also [24]). The method of symplectic cutting that we 
define in this section is different from the one of Woodward. 

We have two actions of K on Xp : the action from the left (resp. right) , denoted 
(resp. v), with moment map : Xp 6* (resp. $r). We consider now the 
product of M X Xp with 

• the action k -i (m, x) = {k ■ m, k > x) : the corresponding moment map is 
$i(m,x) = $m(™) + *r(a;), 

• the action k -2 (m, x) — (m, k -i x) : the corresponding moment map is 
$2(to,x) = $((x). 

Definition 3.16. We denote Mp the symplectic reduction at of M x Xp for the 

action -i ; Mp := {<Pi)-^{0)/ K . 

Note that Mp is compact when $m is proper. The action -2 on M x Xp induces 
an action of K on Mp. The moment map $2 induces an equivariant map $a/p ■ 
Mp t*. Let Z C (<i>i)~-^(0) be the set of points where {K,-i) as a trivial 
stabilizer. 

Definition 3.17. We denote Mp the quotient Z/K C Mp. 
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AI'p is an open subset of smooth points of Alp which is invariant under the in- 
action. The symplectic structure of M x Xp induces a canonical symplectic structure 
on M'p that we denote flj^p . The action of K on {Mp,^lM' ) is Hamiltonian with 
moment map equal to the restriction of ■ Mp i* to M'p. 

We start with the easy 

Lemma 3.18. The image of $_A/p : Mp — > t* is equal to the intersection of the 
image of $m : A/ — » £* with K ■ P. 

Let Up — K ■ Interior(P) C K ■ P. We will show now that the open and 
dense subset ($Afj=)~^(Wp) of Mp belongs to M'p. Afterwards we will prove that 
^]J^{Up) is quasi-symplectomorphic to the open subset ^^.j{L{p) of M. 

We consider the open and dense subset of Xp which is equal to the open orbit 
p{Kc). From Lemma l3.3| we know that 

(3.28) e-.Kxi — > p{Kc) 

ik,X) ^ [p(fce'^)] 

is a diffeomorphism. Through O, the action oi K x K on K xi is k-i{a, X) = {ka, X) 
for the action "from the left" and k > (a, X) = (ak^^, k ■ X) for the action "from 
the right". 

We consider now the map ipK : t ^ t* defined by tpK{X) ~ $/([p(e*'^)]). In 
other words, 



Tr(p(e»2^)) ■ 

Consider the function Fk : t -> K, ^ ln(Tr(p(e-*^)). Let Lk : i i* 

be its Legendre transform. 

Proposition 3.19. • We have ^k{X) = LKi~2X), for X £ t, 

• The function Fj^ is strictly convex, 

• The map ipx realizes an equivariant diffeomorphism between 6 and Up . 

• The image of : Xp i* is equal to the closure of Up, 

• <^l\Up)^p{Kc). 

Proof. For X, F G t we consider the function r(s) = Fk{X + sY). Since 
Fk is -fC-invariant we can restrict our computation to X G t. We will use the 
decomposition of F G 6 relatively to the T- weights on 6c ^ ^ — ^a^a where 
a.(i{Z)Ya = ia{Z)Ya for any Z G t, and Yq G t. We have 

'giad(X,) _ I 

-TT{p{e-^^^)p{Y)) 



-I 



Tr(p(e-^^=)) 



^ {7TKMe''''')),Y) 



Tr(p(e-'^0) 

where Xg = X + sY. Since by definition t'(0) = {Lk{X),Y) , the first point is 
proved. For the second derivative we have 

^ / Tr(p(e-^)p(^y)) y Tr (p(e-^)p(^^:g^^y)p(^y)) 
^ ^ ' \ Tr(p(e-*^)) J Tr(p(e-^^)) 



18 



PAUL-EMILE PARADAN 



where 

^ Tr (p(e-'^)p(»yo)p(»yo)) / Tr(p(e-^)p(»yo)) ^ 
Tr(p(e-^)) Tr(p(e-^)) J 

and 

It is now easy to see that Ri and R2 are positive and that R1+R2 > if y ^ 0. We 
have proved that Fk is strictly convex, So, its Legendre transform Lk reahzes a 
diffcomorphism of i onto its image. Using the first point we know that ipK reahzes 
a diffeoniorphisni of t onto its image. The map ipK is equi variant and coincides 
with tpT on t. We have proved in Proposition 13. 101 that the image of V't is equal to 
the interior of P, hence the image of ipx is l^p- 
For the last two points we first remark that 

(3.29) $,([p(fce^^)]) = fc-^x(X) 

hence the image of is the closure of Up. If we use the fact that ipK is a diffeo- 
niorphisni from t onto shows that ^^^{K ■ ^) n p{Kc) is a non empty and 
closed subset of ^^^i^^ ' for any ^ G Up (in fact it is a iiT x if-orbit). On the 
other hand ^^^{K ■ ^) n {Xp \ p{Kc)) is also a closed subset of $;^^(if • ^) since 
p{Kc)) is open in Xp. Since ^^^i^ ' is connected the second subset is empty : 
in other words ^f^{K ■ ^) C p{Kc)- □ 

We introduce now the equivariant diffcomorphism 

(3.30) T -.K xUp — > p{Kc) 

(fc,C) ^ e(fc,^^i(0). 

We now look at if x Up equipped with the symplectic structure T*{^Xp), and the 
Hamiltonian action oi K x K : the moment maps satisfy 

(3.31) T*{<^i){k,i)^k-i and T*(<i>,)(fc, ^ = 
Proposition 3.20. We have 

T*{flxp)=dX + dr] 

where A is the Liouville 1-form on K x T*K and rj is an invariant I- form on 
Up d t* which is killed by the vectors tangent to the K -orbits. 

Proof. Let Ei, . . . , Er he a, basis of i, with dual basis . . . , Let cj' the 1- 
form on K , invariant by left translation and equal to at the identity. The Liouville 
1-form is X = - ® E^. For X e J wc denote Xi{k, ^) = ^|oe"*^ (fc, and 

Xr{k,^) = ^|oe^*"^ > (fcj?) the vectors fields generated by the action of K x K. 
Since i{Xi)d\ = — and i{Xr)dX = —d{^r,X), the closed invariant 2-form 
/3 = T*{nxp) — dX is K X K invariant and is killed by the vectors tangent to the 
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orbits : (*) i{Xi)l3 ^ t(X,.)/3 ^ for all X e t We have /3 = /32 + /3i + /3o where 
P2 = J2i j ^ij iO^^ ^ J Pi — J2i j ^ij (C)"^* ^ d,Ej , and /3o is an invariant 2-form on 
Up. The equalities (*) gives l{Xi)(32 — i{Xi)(3i = which implies that /?2 = /3i = 0. 
So /3 = /3o is a closed invariant 2-form on Up which is killed by the vectors tangent 
to the X-orbits. Since Up admits a retraction to {0}, (3 — dr] where 77 is an invariant 

1- form on Up which is killed by the vectors tangent to the X-orbits. □ 

If {m,x) E M X Xp belongs to $]^^(0), we denote [m.x] the corresponding 
element in Mp. By definition we have $Mj=(["^, x]) — $/(a;) for [m, x] S Mp, hence 
the image of $Mjj is included in the closure of Up. We see also that [m,x\ € 
^Ij^iUp) if and only if x € ^^^{Up) = p{Kc). Since {K, >) acts freely on p{Kc), 
we see that {K, -i) acts freely on ^Jj^ (^p) '■ the open and dense set i^p) C Mp 
is then contained in Mp. 

Now, we can state our main result which compares the open invariant subsets 
^^j{Up) C M and ^J,^^{Up) C Mp equipped respectively with the symplectic 
structures and 51 • 

Theorem 3.21. ^ll^{Up) is an open and dense subset of smooth points in Mp . 
There exist an equivariant diffeomorphism 5" : <I>^/(Z//p) <^^l^{Up) such that 

Here rj is an invariant l-form on Up which is killed by the vectors tangent to the 
K -orbits. Moreover the path f2* = Qm ^td^AiV^ defines an homotopy of symplectic 

2- forms between o,nd 'I'*(f2jv/^). 

Remark 3.22. The map ^' will be call a quasi- symplectomorphism. 
Proof. Consider the immersion 

^p-.^lliUp) MxXp 

m I — > (m, T(e, $m("^)))- 

We have ^i{^{m)) $m("i) + T*$r(e, <i>M('7i)) = 0, and $2('(/'(m)) = 
T*$,(e,$Af(m)) = $M(m) e Up (see (P3T|) ). Hence for all m e $^/(Z^p), we 
have ?/'(to) e $j"^(0), and its class [ip{m)] G Mp belongs to ^]^j^{Up). 

We denote ^ : <^m{Up) ^li^iUp) the map TO 1-^ [■(/'(to)]. Let us show that it 
defines a diffeomorphism. If '^{m) — 'i'{m/), there exists k E K such that 

(TO,T(e,$M(m))) = fc-i (TO',T(e,$A/(m'))) 

= (fc-m',fc-,T(e,$M(m'))) 

= (fc-m',T(fc-\fc-$M(m'))). 

Since T is a diffeomorphism, we must have k = e and m — m' : the map ^' is one 
to one. Consider now {m,x) £ ^^^{O) such that $Afj> ([w, a;]) = $/(a;) G Z/^p : then 
X e $,"^(i^p) = p(ii:c) = Image(T). We have x = T(fc,^) where C = -$, (a;) = 
<i>A/(TO). Finally 

(m, x) = (m, T(fc, $A/(m))) 

= fc"^ -1 (fc-TO, T(e,/c-$Af(m))) 
= k~^ -1 ?/'(/c • m). 
We have proved that "if is onto. 
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In order to show that 5* is a submersion wc must show that for m G <i>^,j (^p) 

Image(T™V) © T^,(™)(K -i V(m)) = T^,(„.)$r'(0). 

Here Tmip '■ T„iM T^(„j)(Af x Xp) is the tangent map, and T^(^)(i4r-i-0(m)) de- 
notes the tangent space at ip{m) of the {K, •i)-orbit. We have dim(Image(Tm-0)) + 
dim(T^,(„j)(X -1 ip{m))) — dim(T^(m)$j^ (0)) so it is sufficient to prove that 

Image(T™^) n T^(™)(i^ -i ^(m)) = {0}. 

Consider {v,w) G Image(T,„V) ^ T'tij{m){K -i ^{m)). There exists X ^ I such 

V = -J- e*'^ ■ m and w = e*^ > T(e, $m(to)) 
In the other hand since (v^w) £ Image(T.m^), we have 

w = ^ T(e,<i>M(e*^-m)) 

at |o 

Since e*^ > T(e, $a/(to)) = T(e-*-^, $M(e*^ • m)) we obtain that 

1^ <i>M(e*^ . m)) = $M(e*^ • m)) 

or in other words ^|^T(e^*^, $j\f (m)) = 0. Since T is a diffeomorphism we have 
X = 0, and then {v, w) = 0. 

We can now compute the puU-back by of the symplectic form f^M^ ■ We have 

It remains to prove that for every t G [0, 1], the 2-form fi* = JIm + td^XiV is 
non-degenerate. Take t ^ 0, m d ^~^{L{p) and suppose that the contraction of 
by ?; G T^M is equal to 0. For every X G t we have 

= n\XM{m),v) 

= -L{v)d{<^M,X)^^^+tL{v)L{XM)d<S>lif]i^ 

= -/.(t;)d($M,^)|„ 

since L{XM)d<^\jri = d^l^{i,{Xi*)rj) = 0. Thus we have Tm^Miv) = 0, and then 
t{v)d^*j^^ri = 0. Finally we have that — t(?;)r2*|m = L{v)^lM\m■ But JIm is 
non-degenerate, so w = 0. □. 

3.5. Formal quantization : second definition. We suppose here that the Hamil- 
tonian i^-manifold (M, Om, $m) is proper and admits a Kostant-Souriau line bun- 
dle L. Now we consider the complex K x iiT-submanifold Xp of V{E). Since 0{—l) 
is a X if-equivariant Kostant-Souriau line bundle on the projective space F{E) 
the restriction 

(3.32) Lp = 0{-l)U, 

is a Kostant-Souriau line bundle on Xp . Hence LMLp is a. Kostant line bundle on 
the product M x Xp. In section [2?2l we have have defined the quantization Qk{AIp) 
of the (singular) reduced space Mp := {M x Xp)//o{K, -i). 
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Notation: OKir) will be any element X^^eK "^M^/f of ^ °°(iir) where = 
if < ?'. The limit limr^+oo Oa'(t') = defines the notion of convergence in 

Proposition 3.23. Let ep > be the radius of the biggest ball center at E t* 
which is contains in the polytope P. We have 

(3.33) Qk(Mp)= J2 + Oxiep). 

\\fi\\<ep 

Proof. Theorem 12.41 - "Quantization commutes with reduction in the singu- 
lar setting" - tells us that Qk{Mp) = J2f,eK Qii^-^p)t^)^i!^ where (Mp)^ is the 
symplectic reduction 

(Mp X K^)/hK2 ^ [M xXpy. X^)//(o,o)^2 x Ki. 

Recall what the A'l, _R'2-action are: k -i (m, x, ^) = {km, k > ^) and k -2 (to, x, ^) = 
(to, k -i x, k^) for (to, x, ^) G Af x Xp x K2 ■ fJ, and k £ K. 

Since the image of is equal to the intersection oi K ■ P = Up with the image 
of <^M, we have 

(3.34) Q((Mp)^)=0 if M^^'nlmage($M). 

We will now exploit Theorem 13.211 to show that Q{{Mp)^) ~ Q{M^j) if /i belongs 
to the interior of P. 

There exists a quasi-symplectomorphism ^ between the open subset ^J.j{Up) of 
M and the open and dense subset <i>^/ (Hp) of Mp. Moreover one can see easily 
that the restriction of the Kostant line bundle Lp ^ Xp to the open subset p{Kc) 
is trivial. If Lmp is the Kostant line bundle on Mp induced by LMLp, we have that 
the pull-back of the restriction -^a/p by ^' is equivariantly diffeomorphic to 

the restriction of L to ^Jj{Llp). 

Take now ji £ K that belongs to the interior of the polytope P. The element 
Q((Mp)p) G Z is given by the index of a transversally elliptic symbol defined in 
a (small) neighborhood of {p) C Mp. This symbol is defined through two 
auxiliary data: the Kostant line bundle and a compatible almost complex 

structure J which defined in a neighborhood of (/i). If we pull back everything 
by we get a transversally elliptic symbol living in a (small) neighborhood of 
(m) -^^ which is defined by the Kostant line bundle L and an almost complex 
structure Ji compatible with the symplectic structure ili :— U,m + d^^jrj. But 
since VLt = + td^MV defines an homotopy of symplectic structures, any almost 
complex structure compatible with JIm is homotopic to Ji. We have then shown 
that Q(Afp) — Q{{Mp)f^) for any ^ belonging to the interior of P. So we have 

Qk{Mp) = 2(M^)t/^^ -I- ^ Q((Afp),)I/^. 

/^elntcrior(P) i/edP 

Since for v G dP we have > ep, the last equality proves p.33p . □ 

We work now with the dilated polytope nP, for any integer n > 1. The polytope 
nP is still iiT-adapted, so one can consider the reduced spaced M„p and Proposition 
13.231 gives that 

(3.35) Qk(M„p) = J2 2(^'^A<)^/f + Oxinep). 

\\fj,\\<nEp 
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for any integer n > 1. We can summarize the result of this section in the following 

Proposition 3.24. Let {M, ^m) be a pre- quantized Hamitonian K -manifold, with 
a proper moment map $m- 

• For any integer n > I, the (singular) compact Hamiltonian manifold MnP 
contains as an open and dense subset, the open subset ^J,j{nUp) of M . 

• We have Q^°°(M) = lim Qk{M„p). 

n — >cx} 

4. FUNCTORIAL PROPERTIES : PROOF OF THEOREM 11.31 



This section is devoted to the proof of Theorem 11.31 We will used in a crucial 
way the characterisation of Qx°° given in Proposition 13 . 24l 

Let H d K he a connected Lie subgroup. Here we consider a pre-quantized 
Hamiltonian X-manifold M which is proper as an Hamiltonian _ff-manifold. We 
want to compare Q^°°(M) and Q^°°(M). For fj, £ K and v e H we denote N^f the 
multiplicity of in the restriction V^\h- We have seen in the introduction that 
Nl}Q {M^,^k) / only for the ^ belonging to finite subset X n $k • ^h^{v)). 
Then Q~^{M) is i/-admissible and we have the following equality in R^°°{H) : 

(4.36) Q-~(M)|h = J^m.K'' 

with = NlfQ{M^,.K). We wiU now prove that Qk°°{M)\h = Qh°°{M). 
Lemma 4.1. The restriction QfJ^ {M)\h is equal to lim Q7<(M„p)|//. 

n — >oo 

Proof. Let us denote P° and dP respectively the interior and the border of 
the /^-adapted polytope P. We write 

On the other side 

Qk(M„p)= E QiM^M)V^'+ E QiiMnp)f.,K)V^^. 
fi^nP° ^^jidP 

So the difference D{n) = Q]^°^\M) - Qk(M„p) is equal to 

D{n) = - E Qmnp)^',K)V^^ + E QiM^.K)V^- 

fj,'£ndP fifnP° 

We show now that the restriction D{n)\H tends to in R^°^{H) as n goes to 
infinity. For this purpose, we will prove that for any c > there exist Uc (z N such 
that D{n)\H — Oh{c) for any n > Uc- 

For c > we consider the compact subset of i* defined by 

(4.37) /Cc = $K (if • e t)Mieil < c)) . 

Let Uc € N such that ICc is included in K ■ {ucP") : hence K-c C K ■ {nP°) 
for any n > Uc- We know that for fi ^ K, we have NjfQ{M^K) 7^ only for 
^l ^ <^K {K ■<^-^{v)), and for ^' £ K, we have Nff' Q{{Mnp)t,' ,k) / only for 

n' enPn<i>K (if • $^^(i^)). 
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Then if n > ric, we have 

for any v e H {£, e t}* \\^\\ < c}, ^ ^ nP° and /i' G n9P. It means that 
D{n)\H = Oh{c) for any n > ric. □ 

Since Qk(M„p)|h = QniMnp), we are no led to the 
Lemma 4.2. T/ie Um Qh(M„p) is egwa^ <o Q^°°(M). 

n — >oo 

Proof. Theorem 12.41 - "Quantization commutes with reduction in the singular 
setting" - tells us that Qi/(M„p) = J^ueS Q{{Mnp)uM)V^ where {Mnp)^,H is the 
symplectic reduction 

(M„p X Tn^)//oH = (M X X„p X irT7I)//(o,o)if X K. 

For c > we consider the compact subset of ICc defined in (|4.37p . Let ric E N 
such that ICc C K ■ {nP°) for any n > ric- It implies that 

(e e t)M|^|| < c) c ■ (^n) 

for n > Tic- Since Af„p "contains" as an the open subset ^~^^{K ■ {nP°)), the argu- 
ments similar to those used in the proof of Proposition [3?23l show that Q{{Mnp)i^.H) = 
Q{M^ h) for \\v\\ < c and n > n^,. It means that 

Qh(M„p)= ^ Q{M,,h)V," + Oh{c) when n > n,. 

Ikil<c 

It follows that lim (Af„p) = V - = Q^°°(Af). □ 

5. The case of an Hermitian space 
Let (E, h) be an Hermitian vector space of dimension n. 

5.1. The quantization of E. Let U := V{E) be the unitary group with Lie 
algebra u. We use the isomorphism e : u ^ u* defined by {e{X),Y) = —Tr(XY) £ 
M. For v,w E E, let V (E) w* : E —^ E he the linear map x h(a;, w)v. 

Let Er be the space E view as a real vector space. Let ft be the imaginary 
part of — h, and let J the complex structure on Sr. Then on E]^, f2 is a (constant) 
symplectic structure and ri(— , J— ) defines a scalar product. The action of U on 
{Er,^!) is Hamiltonian with moment map ^ : E ^ u* defined by ($(«), X) = 
■^n{Xv, v). Through e, the moment map $ is defined by 

(5.38) ^{v)^-^v®v*. 

2i 

The pre-quantization data (L, (—,—), V) on the Hamiltonian U-manifold 
(i?R, ri, $) is a trivial line bundle L with a trivial action of U equipped with the 
Hermitian structure (s, 5'}^, — e ss' and the Hermitian connexion V — d — i9 
where 6 is the 1-form on E defined by 61 = ^n{v,dv). 

The traditional quantization of the Hamiltonian U-manifold (£'r, f], <I>), that 
we denote (E), is the Bargman space of entire holomorphic functions on E 
which are L^ integrable with respect to the Gaussian measure e ^^fi". The 
representation Qj) {E) of U is admissible. The irreducible representations of U 
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that occur in Q]j (E) are the vector subspaces S^{E*) formed by the homogeneous 
polynomial on E of degree j > 0. 

On the other hand, the moment map $ is proper (see I5.38p . Hence we can 
consider the formal quantization Q^°°{E) G R~°°(\J) of the U-action on E. 

Lemma 5.1. The two quantizations of (£',17,$), Q}} (E) and Q^°°{E) coincide 
in i?^°°(U). In other words, we have 

(5.39) Q^°^iE) ^ S'iE*) -.^^J^^^i^*) ill R'^iU). 

Proof. Let T C U be a maximal torus with Lie algebra t C u. There exists 
an orthonormal basis {ek)k=i,--- ,n of E and characters (xfc)/c=i,- - ,n of T such that 
t ■ Sk = Xk{t)ek for all k. The family {ick ^ e^)k=i,--- ,n is then a basis of t such 
that jdxiiisk <8> e^) — Si_k- The set U C t* C u* of dominants weights is composed, 
through e, by the elements 

n 

X = i^XkSk ® el, 

k=l 

where A — (Ai, A2, • • • , A„) is a decreasing sequence of integer. 
The formal quantization Q^°°{E) e i?"°°(U) is defined by 



Ai>->A„ 



where E\ = ^~^{U ■ X)/U is the reduced space and V\ is the irreducible represen- 
tation of U with highest weight A. 
It is now easy to check that 



Ex = 



{pt} if A=(0,---,0,-j)withj>0, 
in the other cases, 



and then 



1 if A = (0,--- ,0,-j) withj>0, 
in the other cases. 



QiEx) 

Finally ([05)) follows from the fact that V(o,... .o,-j) = S^iE*). □ 



5.2. The quantization of E restricted to a subgroup of U. Let C U be a 

connected Lie subgroup with Lie algebra 6*. Let Kc C GL{E) be its complexifica- 
tion. The moment map relative to the if-action on (£r, fl) is the map 

$K : £ r 

equal to the composition of $ with the projection u* t* . 

Lemma 5.2. The following conditions are equivalent : 

(a) the map is proper, 

(b) <^-^}[Q) = {Q}, 

(c) {0} is the only closed Kc-orbit in E, 

(d) for every v ^ E we have € Kc ■ v, 

(e) S*{E*) is an admissible representation of K , 

(f) the K -invariant polynomials on E are the constant polynomials. 
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Proof. The equivalence (a) <;=^ (6) is due to the fact that is quadratic. 

Let O be a Xc-orbit in E. Classical results of Geometric Invariant Theory [261IT8] 
assert that On $^^(0) ^ and that O is closed if and only if O n $^^(0) ^ 0. 
Hence (6) ^ (c) (d). 

After Lemma O we know that Q^°"{E) = S'{E*). Since Q^°°{E) is in- 
admissible when is proper (see Section |4|), we have (a) =J> (e). 

For every ^ & K, the //-isotopic component [S'*(i?*)]^ is a module over [S"(i?*)]o = 
[S'iE*)]'^. Hence dim[S"(i;*)]^ < oo implies that [S*{E*)]'^ = C. We have 
(e) =^ (/). 

Finally (/) =^ (d) follows from the following fundamental fact. For any v,w ^ E 
we have lOTv n Kc ■ w ^ if and only if P{v) = P{w) for all P G [5"(£;*)]^. □ 

Theorem 11.31 implies the following 

Proposition 5.3. Let K C U(£^) be a closed connected subgroup such that S'{E*) 
is an admissible representation of K . For every fj, G K, we have 

dimi[S'iE*)]f,) = Q{E^,,k) 

where [S'{E*)]|^^ is the fj,-isotopic component of S*{E*) and E^^k is the reduced 
space ^^{K ■ pi)/K. 

In the following examples the condition ^^^^{0) — {0} is easy to check. 

1) the subgroup K C U(£') contains the center of U(i?), 

2) E = A^C" oyE^ 5*2 (C") and K = U(n) C ViE), 

3) E = M„ is the vector space of n x fc-matrices and K = U(n) x U(fc) C 
ViE). 
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